Three dimensional reconstruction of scenes from multiple views using active vision

The need to understand the mechanisms underlying the growth of plants and crops (plant phenotyping) is becoming increasingly important in society, particularly as the quantity of food and biofuel will need to double to meet the demands of the expanding global population, which is likely to exceed nine billion by 2050. The practical aim of this research is to contribute to reducing the bottleneck associated with plant phenotyping by generating a fully automated response to photometric data acquisition and the recovery of three-dimensional models of plants from multiple views without the dependency of botanical expertise, ensuring a non-intrusive and non-destructive approach. Plants are complex objects displaying high degrees of concavity and self-occlusion and can be considered examples of crowded scenes. This unavoidable complexity makes careful camera placement a necessity. If a complete 3D reconstruction is to be achieved, viewpoints must be chosen to reflect the broad 3D structure of the plant. Within this thesis, an Active Vision Cell (AVC), consisting of a camera-mounted robot arm, turntable and automatic image acquisition technique is proposed, along with a novel surface reconstruction algorithm. This approach provides a robust, flexible and accurate approach to automating 3D reconstruction of plants. The active vision method exploits volumetric shape representations to provide a compact image set well-suited to multi-view stereo. The reconstruction algorithm can reduce noise and provides a promising and extendable framework, improving on the current state-of-the-art. Furthermore, the pipeline can be applied to any plant species or form due to its application of an active vision framework combined with the automatic detection of key parameters for surface reconstruction

Active Vision and Surface Reconstruction for 3D Plant Shoot Modelling

Plant phenotyping is the quantitative description of a plant’s physiological, biochemical and anatomical status which can be used in trait selection and helps to provide mechanisms to link underlying genetics with yield. Here, an active vision- based pipeline is presented which aims to contribute to reducing the bottleneck associated with phenotyping of architectural traits. The pipeline provides a fully automated response to photometric data acquisition and the recovery of three-dimensional (3D) models of plants without the dependency of botanical expertise, whilst ensuring a non-intrusive and non-destructive approach. Access to complete and accurate 3D models of plants supports computation of a wide variety of structural measurements. An Active Vision Cell (AVC) consisting of a camera-mounted robot arm plus combined software interface and a novel surface reconstruction algorithm is proposed. This pipeline provides a robust, flexible and accurate method for automating the 3D reconstruction of plants. The reconstruction algorithm can reduce noise and provides a promising and extendable framework for high throughput phenotyping, improving current state-of-the-art methods. Furthermore, the pipeline can be applied to any plant species or form due to the application of an active vision framework combined with the automatic selection of key parameters for surface reconstruction

A canopy conundrum: can wind-induced movement help to increase crop productivity by relieving photosynthetic limitations?

Wind-induced movement is a ubiquitous occurrence for all plants grown in natural or agricultural settings and in the context of high, damaging wind speeds it has been well studied. However, the impact of lower wind speeds (that do not cause any damage) on mode of movement, light transmission and photosynthetic properties has, surprisingly, not been fully explored. This is likely to be influenced by biomechanical properties and architectural features of the plant and canopy. A limited number of eco-physiological studies have indicated that movement in wind has the potential to alter light distribution within canopies, improving canopy productivity by relieving photosynthetic limitations. Given the current interest in canopy photosynthesis is timely to consider such movement in terms of crop yield progress. This opinion article sets out the background to wind-induced crop movement and argues that plant biomechanical properties may have a role in the optimisation of whole canopy photosynthesis via established physiological processes. We discuss how this could be achieved using canopy models

Recovering Wind-induced Plant motion in Dense Field Environments via Deep Learning and Multiple Object Tracking

Understanding the relationships between local environmental conditions and plant structure and function is critical for both fundamental science and for improving the performance of crops in field settings. Wind-induced plant motion is important in most agricultural systems, yet the complexity of the field environment means that it remained understudied. Despite the ready availability of image sequences showing plant motion, the cultivation of crop plants in dense field stands makes it difficult to detect features and characterize their general movement traits. Here, we present a robust method for characterizing motion in field-grown wheat plants (Triticum aestivum) from time-ordered sequences of red, green and blue (RGB) images. A series of crops and augmentations was applied to a dataset of 290 collected and annotated images of ear tips to increase variation and resolution when training a convolutional neural network. This approach enables wheat ears to be detected in the field without the need for camera calibration or a fixed imaging position. Videos of wheat plants moving in the wind were also collected and split into their component frames. Ear tips were detected using the trained network, then tracked between frames using a probabilistic tracking algorithm to approximate movement. These data can be used to characterize key movement traits, such as periodicity, and obtain more detailed static plant properties to assess plant structure and function in the field. Automated data extraction may be possible for informing lodging models, breeding programmes and linking movement properties to canopy light distributions and dynamic light fluctuation

Approaches to three-dimensional reconstruction of plant shoot topology and geometry

There are currently 805 million people classified as chronically undernourished, and yet the World’s population is still increasing. At the same time, global warming is causing more frequent and severe flooding and drought, thus destroying crops and reducing the amount of land available for agriculture. Recent studies show that without crop climate adaption, crop productivity will deteriorate. With access to 3D models of real plants it is possible to acquire detailed morphological and gross developmental data that can be used to study their ecophysiology, leading to an increase in crop yield and stability across hostile and changing environments. Here we review approaches to the reconstruction of 3D models of plant shoots from image data, consider current applications in plant and crop science, and identify remaining challenges. We conclude that although phenotyping is receiving an increasing amount of attention – particularly from computer vision researchers – and numerous vision approaches have been proposed, it still remains a highly interactive process. An automated system capable of producing 3D models of plants would significantly aid phenotyping practice, increasing accuracy and repeatability of measurements

A canopy conundrum: can wind-induced movement help to increase crop productivity by relieving photosynthetic limitations?

Wind-induced movement is a ubiquitous occurrence for all plants grown in natural or agricultural settings and in the context of high, damaging wind speeds it has been well studied. However, the impact of lower wind speeds (that do not cause any damage) on mode of movement, light transmission and photosynthetic properties has, surprisingly, not been fully explored. This is likely to be influenced by biomechanical properties and architectural features of the plant and canopy. A limited number of eco-physiological studies have indicated that movement in wind has the potential to alter light distribution within canopies, improving canopy productivity by relieving photosynthetic limitations. Given the current interest in canopy photosynthesis is timely to consider such movement in terms of crop yield progress. This opinion article sets out the background to wind-induced crop movement and argues that plant biomechanical properties may have a role in the optimisation of whole canopy photosynthesis via established physiological processes. We discuss how this could be achieved using canopy models

Three-dimensional reconstruction of plant shoots from multiple images using an active vision system

The reconstruction of 3D models of plant shoots is a challenging problem central to the emerging discipline of plant phenomics – the quantitative measurement of plant structure and function. Current approaches are, however, often limited by the use of static cameras. We propose an automated active phenotyping cell to reconstruct plant shoots from multiple images using a turntable capable of rotating 360 degrees and camera mounted robot arm. To overcome the problem of static camera positions we develop an algorithm capable of analysing the environment and determining viewpoints from which to capture initial images suitable for use by a structure from motion technique

A Deep Learning Method for Fully Automatic Stomatal Morphometry and Maximal Conductance Estimation

Stomata are integral to plant performance, enabling the exchange of gases between the atmosphere and the plant. The anatomy of stomata influences conductance properties with the maximal conductance rate, gsmax, calculated from density and size. However, current calculations of stomatal dimensions are performed manually, which are time-consuming and error prone. Here, we show how automated morphometry from leaf impressions can predict a functional property: the anatomical gsmax. A deep learning network was derived to preserve stomatal morphometry via semantic segmentation. This forms part of an automated pipeline to measure stomata traits for the estimation of anatomical gsmax. The proposed pipeline achieves accuracy of 100% for the distinction (wheat vs. poplar) and detection of stomata in both datasets. The automated deep learning-based method gave estimates for gsmax within 3.8 and 1.9% of those values manually calculated from an expert for a wheat and poplar dataset, respectively. Semantic segmentation provides a rapid and repeatable method for the estimation of anatomical gsmax from microscopic images of leaf impressions. This advanced method provides a step toward reducing the bottleneck associated with plant phenotyping approaches and will provide a rapid method to assess gas fluxes in plants based on stomata morphometry

Plant phenotyping: an active vision cell for three-dimensional plant shoot reconstruction

Three-dimensional (3D) computer-generated models of plants are urgently needed to support both phenotyping and simulation-based studies such as photosynthesis modelling. However, the construction of accurate 3D plant models is challenging as plants are complex objects with an intricate leaf structure, often consisting of thin and highly reflective surfaces that vary in shape and size, forming dense, complex, crowded scenes. We address these issues within an image-based method by taking an active vision approach, one that investigates the scene to intelligently capture images, to image acquisition. Rather than use the same camera positions for all plants, our technique is to acquire the images needed to reconstruct the target plant, tuning camera placement to match the plant’s individual structure. Our method also combines volumetric- and surface-based reconstruction methods and determines the necessary images based on the analysis of voxel clusters. We describe a fully automatic plant modelling/phenotyping cell (or module) comprising a six-axis robot and a high-precision turntable. By using a standard colour camera, we overcome the difficulties associated with laser-based plant reconstruction methods. The 3D models produced are compared with those obtained from fixed cameras and evaluated by comparison with data obtained by X-ray μ-computed tomography across different plant structures. Our results show that our method is successful in improving the accuracy and quality of data obtained from a variety of plant types